EnigmA Amiga Run 1998 July/EnigmA AMIGA RUN 29 (1998)(G.R. Edizioni)(IT)[!][issue 1998-07 & 08].iso/recent/urlupd.lha/URLUpDat/ReadMe.txt

Text File | 1998-05-18 | 14KB | 234 lines

URLUpDat V1.00.19980518. © 1998 Michael J. Maloney BSc (Hons). Date Monday
18th May 1998.

Syntax: URLUpDat FILE_SOURCE_DIRECTORY FILE_DESTINATION_DIRECTORY
URL_INPUT_FILE URL_OUTPUT_FILE URL_MODIFIERS_FILE
BEGIN_HOURS BEGIN_MINUTES BEGIN_SECONDS PAUSE_SECONDS
FAILED_URLS_FILE MOVE_FILES_SHELL_SCRIPT

This program will not download files directly from the internet. It scans a
list in ascii format containing the Universal Resource Location (internet
address) of all the files that you would normally download over a given
period of time, moves the relevant downloaded files to a different location
altering their filenames and datestamps to user specified values, before
updating the original URL list. The updated list of URLs can then be passed
to whichever download utility you are using so that it can download the
next batch of files from the internet. The download utility that I use for
this purpose is Filehound on the PC, available from
'http://www.windows95.com/apps/98/utilities.html' (or thereabouts at the
time of writing). It is shareware, small, simple to use, and will not cripple
you if you do not register it. A similar Amiga program should be available
from Aminet 'http://www/cucug.org/aminet.html'. The operation of
URLUpDat can be best explained by covering the arguments used in its
operation:

FILE_SOURCE_DIRECTORY e.g. URLUpDat:FilesIn/
This is the source directory from where URLUpDat will look for downloaded
files. I recommend that you backup this before running URLUpDat as once a
downloaded file has been copied to its destination directory and any
alterations to its filename and datestamp have been successfully completed
then the original is no longer needed and automatically deleted.

FILE_DESTINATION_DIRECTORY e.g. URLUpDat:FilesOut/
This is the destination directory where URLUpDat will copy any modified
files. WARNING: as it stands the Amiga Dos Copy command will not check if
there is enough room for the copied files on the destination volume. It is
incumbent on the user to do so. This should not be a problem as URLUpDat
only copies one file at a time, but those of you out there (including myself)
who use their Amigas in conjunction with their PCs via the shareware
program PC2Amiga (available from Aminet 'comms/net/PC2Am308.lha') will
have to make sure that there is enough room for all the copied files on
the Amiga if URLUpDat is copying them directly from the PC. Failure to do
so will result in checksum errors on the Amiga destination volume once it
has run out of room, and several hours spent repairing the damage with
DiskSalv.

URL_INPUT_FILE e.g. Ram:URL_In.txt
This is a simple ascii file containing a list of the URLs that you would like
to download in the order that you would like to downloaded them. NB. Many
download utilities download files in parallel. That way if a web site is very
slow the other files do not have to be stuck in a queue waiting for one
stubborn file to finish downloading. This is great, but does tend to
randomize the file order if you are listing them chronologically. URLUpDat
sets the datestamps of selected files so that they reflect the date and
time according to the position that they appear on this list, and not the
date and time that the file was actually downloaded.

Each URL should be on a separate line. URLUpdat will scan backwards from
the end of this line until it finds a '/' character, signifying the division
between the location of the file and the filename itself. For this reason all
valid lines must contain a '/' somewhere on them. The Filename itself is
then subdivided into an alpha, a numeric and an extension section for
processing e.g. 'file', '999', '.txt'. URLUpDat assumes that most regularly
updated files will similarly be comprised of an alpha section, followed by a
numeric section followed by an extension; filenames that jumble their
alphanumeric characters may not be processed correctly by URLUpDat.

The first time that URLUpDat encounters '/Monday' it will prompt the user
to enter the date of the first Monday from which file downloading was
meant to begin (rarely the same as the date that it actually did). The date
is entered as a single eight digit number in the form year (4), month (2),
day (2) e.g. 19980618. Those using the American system for dates take
special care to follow the European convention here; a derivative of this
number can later be incorporated into the modified filename allowing files
with the same alpha prefix to be listed in the same order whether they
are listed alphabetically or chronologically. This does not happen with the
US date convention as the month and day are taken out of the years,
months, days, hours, minutes and seconds chronological sequence. Following
on from the first Monday every time that URLUpDat encounters a valid day
it will increase by one the date of files that were supposed to be
downloaded on that day, resetting the time value to its default for a new
day. The example script is configured for all the files that might be
downloaded in one week (this is the time scale that the majority of web
masters use when updating their sites), but it could be configured for one
day or one month or more, there is no upper limit, as long as no days are
skipped. URLs encountered before the first '/Monday' will retain there
original datestamps, even if their filenames are altered.

I recommend backing up this file as URLUpDat will probably automatically
overwrite it with an updated list of URLs once all modifications to the
current crop of downloaded files are completed successfully cf.
URL_OUTPUT_FILE.

URL_OUTPUT_FILE e.g. Ram:URL_Out.txt
Once all the downloaded files from the original ascii list of URLs have been
processed then an updated list of URLs is prepared and saved as an ascii
file. This argument should normally have the same value as the
URL_INPUT_FILE argument, ensuring that the old list of URLs is overwritten
after it has served its purpose, but here and in the IconX example script
both files are held separately in RAM: before the old list of URLs is
overwritten by the new.

URL_MODIFIERS_FILE e.g. Ram:URL_Mods.txt
This is a standard delimited ascii file containing the desired modifications
to the downloaded files and the URL list that was used to download them.
Discrete data segments are separated by commas; strings are enclosed
within double quotation marks. Every line holds a separate modification
instruction. A typical modification instruction is broken down into five
elements:
"http://www.files.com/","file",7,"new","date"
The first part of the instruction outlines the full internet address of the
downloaded file excluding the filename. E.g. if the full URL were
'http://www.files/file999.txt' all that would be required is
'http://www.files.com/' not the 'file999.txt file' part. If you will it is
the 'directory' containing the file.

The second part is the alpha part of the filename. E.g. if the full URL were
again 'http://www.files.com/file999.txt' the filename would be 'file999.txt'
and all that would be required is 'file'. Only one instruction is needed for
URLs that are identical up to this part in their structure. E.g. in this case
one instruction would process all the URLs between
'http://www.files.com/file1.txt' and 'http://www.files.com/file999.txt' and
beyond.

Upon successfully downloading a file from the ascii list of URLs the
corresponding URL is incremented by a set number. This ensures that when
the updated list of URLs is again fed to a download utility it will not
download the same file again, but the next in sequence. The third part of
the instruction is the number by which the successful URL is incremented
(failed URLs are fed back as are). Setting this value to 0 means that the
URL will not be altered (some files at the end of URLs are changed on a
regular basis even though their URLs are not).

The fourth part of the instruction determines what the alpha part of the
filename of the modified file will be. E.g. suppose the original downloaded
file was called 'file999.txt'. Setting the fourth parameter to 'new' means
that once the file has been processed it will have the filename
'new999.txt'.

By setting the fifth parameter of the instruction to 'date' the modified
file will incorporate the file's date into its modified filename between the
alpha and numeric segments. This makes sure that URLs that are supposed
to remain constant from download to download will nevertheless generate
files with unique filenames that when listed alphabetically will retain the
same characteristics as when listed chronologically. In this case, the
downloaded file 'file999.txt' will be renamed 'new19980518999.txt' assuming
it was supposed to be downloaded on Monday 18th May 1998. If you wish to
disable this option then set the parameter to any other value than 'date';
'no' works quite well.

BEGIN_HOURS e.g. 03
This argument comprises a two digit number between 00 and 23
representing the 24 hour clock. When a new valid day is encountered in the
ascii list of URLs the timestamp of the next file processed is set to the
default value for a new day, determined by this and the following two
arguments. E.g. if this argument is set to 18 and the following two
arguments are set to 30 and 15 then every time the list of URLs calls for a
new day the timestamp of the first file that was supposed to be
downloaded on that day will be set to 18:30:15. NB. Try not to use 00 here;
this is the value that I use and if ever I get any files from someone using
this program it will mess up my file order (hey, if you are not paying for
this program then this is the least that you can do).

BEGIN_MINUTES e.g. 00
This argument comprises a two digit number between 00 and 59
representing the minutes in an hour.

BEGIN_SECONDS e.g. 00
This argument comprises a two digit number between 00 and 59
representing the number of seconds in a minute.

PAUSE_SECONDS e.g. 20
The timestamps of sequentially processed files will be incremented by a
number of seconds equal to the numeric value set here. E.g. if this is set
to 90 then when the processed files are examined it will appear as if
every file took one and a half minutes to download. This allows the later
insertion of files into the 'gaps'. If this argument is set to 0 then the
modified files will retain there original datestamps. A word of caution
before you choose this value. If the interval is so great that it causes the
clock to go from 23:59 to 00:00 the date will not increase, and new files
will have earlier datestamps than older ones. Only a valid day encountered
by URLUpDat in the list of URLs can trigger a date change.

FAILED_URLS_FILE e.g. Ram:URL_Fail.txt
This is an ascii file containing a list of URLs that failed to download (when
it comes time to retry downloading failed URLs depending on which download
utility you use it is easier to feed into it a list of failed URLs rather than
the entire list).

MOVE_FILES_SHELL_SCRIPT e.g. Ram:URL_Move.txt
URLUpDat does not modify files directly. It creates a shell script listing
desired modifications to downloaded files that the user must execute (the
example IconX script does this for you automatically). This argument
specifies the name of the ascii file containing this script. Having a shell
script allows you to see exactly what actions will be carried out on your
downloaded files before they are carried out; it also allows anyone with a
working knowledge of word processors and A-Rexx macros to further
modify the actions to be performed on their downloaded files without an in
depth knowledge of 'C'. A file will only be included in this script if it
already exists in the source directory, matches a valid URL in the ascii
list of URLs and its modified filename would not overwrite an existing file
in the destination directory.

Known bugs: There are none that I have found yet, however, because PCs
put a carriage return as well as a line feed to signal the end of a line in
an ascii file, you should only use scripts that have been prepared on an
Amiga, not a PC. When I was writing this program I initially used the list of
URLs I had prepared using Notepad and Filehound on the PC. When URLUpDat
tried to process this it worked fine the first time, but not the second.
URLs are case sensitive, and they are handled as such by URLUpDat, so
take care when entering information into the URLs and Mods script files.

Disclaimer: The author accepts no liability for any loss or damage incurred
through use or misuse of this program. This program is shareware; if you
find that it spares you from endless tedious hours of keeping your URL
addresses and downloaded files in good order then please spare a thought
for the author and send the equivalent of £10 sterling to the postal
address below. As the author does not believe in crippleware, you will
receive no benefit for doing so other than knowing that you have paid
good money for good service and helped in the development of this and
future products. The Amiga is dead; long live the Amiga.

The example URLs in the Scripts directory are exactly that. The author
does not bear any responsibility for the content at the end of these links,
nor does the author encourage users of this program to follow them.
Users do so at their own risk. There is nothing to stop anyone from
including their own list of URLs with this program.

Mr. M. J. Maloney
53 Vale Road
COLWICK
Nottingham
NG4 2GL
United Kingdom.